Configurable ordering controller for coupling transactions

ABSTRACT

A method for coupling transactions with a configurable ordering controller in a computer system. The method comprises sending, by a coupling device, first data packets with an unordered attribute being set to an ordering controller. The method further comprises sending, by the coupling device, second data packets with requested ordering to the ordering controller, back-to-back after the first data packets, without waiting until all of the first data packets are completed. The method further comprises sending, by the ordering controller, the first data packets to a memory subsystem in a relaxed ordering mode, wherein the ordering controller sends the first data packets to the memory subsystem in an arbitrary order, and wherein the ordering controller sends the second data packets to the memory subsystem after sending all of the first data packets to the memory subsystem.

BACKGROUND

The present invention relates generally to a short distance couplingenvironment, and more particularly to latency reduction in a shortdistance coupling environment.

For short distance coupling, like many other I/O protocols, it needs tobe ensured, from a high level perspective, that a transaction is onlysignaled to be completed after all payloads have arrived. To accomplishthis, ordering mechanisms are applied by the memory subsystem and thecoupling message hardware side. For a coupling device (for example, anintegrated cluster bus (ICB)), a short distance coupling protocol, twoimplementations for accomplishing this goal in current designs are asfollows. First, the coupling device sends in all packets, includingpayload packets and signaling packets, into a memory subsystem andrequests all the packets to be ordered. As a result, an orderingcontroller (for example, a PCI bus controller (PBC)) will ensure thateach individual packet (including each of the payload packets) will beordered. The first implementation results in unnecessary ordering and asignificant throughput bottleneck. Second, the coupling device sends inall payload packets which are unordered and waits until all of thepayload packets are complete; then, the coupling device sends insignaling packets. The second implementation has better throughput thanthe first implementation. The second implementation is a chosen protocolfor IBM z13®.

SUMMARY

A method for coupling transactions with a configurable orderingcontroller in a computer system is provided. The method includessending, by a coupling device, first data packets with an unorderedattribute being set to an ordering controller. The method furtherincludes sending, by the coupling device, second data packets withrequested ordering to the ordering controller, back-to-back after thefirst data packets, without waiting until all of the first data packetsare completed. The method further includes sending, by the orderingcontroller, the first data packets to a memory subsystem in a relaxedordering mode, wherein the ordering controller sends the first datapackets to the memory subsystem in an arbitrary order, and wherein theordering controller sends the second data packets to the memorysubsystem after sending all of the first data packets to the memorysubsystem.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a diagram illustrating a short distance coupling environment,in accordance with one embodiment of the present invention.

FIG. 2 is a diagram illustrating a first current implementation forcoupling transactions, in accordance with an example of the firstcurrent implementation.

FIG. 3 is an example ordering matrix for a first current implementationshown in FIG. 2, in accordance with an example of the first currentimplementation.

FIG. 4 is a diagram illustrating a second current implementation forcoupling transactions, in accordance with an example of the secondcurrent implementation.

FIG. 5 is a diagram illustrating an implementation for couplingtransactions, in accordance with one embodiment of the presentinvention.

FIG. 6 is an example of a relaxed ordering matrix for an implementationshown in FIG. 5, in accordance with one embodiment of the presentinvention.

DETAILED DESCRIPTION

FIG. 1 is a diagram illustrating short distance coupling environment100. Short distance coupling environment 100 includes coupling device101 (for example, an integrated cluster bus (ICB)), ordering controller102 (for example, a PCI bus controller (PBC)), and memory subsystem 103(for example, an level 3 (L3) cache) of a computer system. Orderingcontroller 102 sits in between coupling device 101 (a sender of payloadpackets and signaling packets) and memory subsystem 103 (a receiver ofpayload packets and signaling packets). In an embodiment of the presentinvention, Ordering controller 102 sends in first data packets (forexample, payload packets) and second data packets (for example,signaling packets) back-to-back and reinterprets the unordered attributeof first data packets (for example, payload packets) as relaxedordering. All memory commands with a flag of relaxed ordering are sentunordered towards each other into memory subsystem 103. The flag is usedby coupling device 101 for commands of the first data packets (forexample, the payload packets), in the same way as in the second currentimplementation mentioned previously in this document. However, unlike inthe second current implementation, coupling device 101 does not wait forcompletions of the first data packets (for example, the payloadpackets), and coupling device 101 sends in all the second data packets(for example, the signaling packets), which are without theunordered/relaxed ordering flag being set, back-to-back after the firstdata packets (for example, the payload packets). For these the seconddata packets (for example, the signaling packets), ordering controller102 will make sure that all preceding packets (the first data packets),which are with the relaxed ordering bit set, have completedsuccessfully. Therefore, an operation for the second data packets canonly be seen after all the first data packets have completed. Orderingcontroller 102 includes a mode bit to decide whether an unordered bit ofa certain bus interface shall be treated as truly unordered or as thenew relaxed ordering mechanism. The advantage of the implementation inthe present invention is reduced latency for coupling messages.

The present invention discloses an implementation in the nest orderinglogic to optimize the performance of coupling messages while stillguaranteeing the required ordering. An embodiment of the presentinvention introduces a relaxed ordering matrix mode for the first datapackets (for example, the payload packets). In the treatment ofunordered as relaxed ordering in ordering controller 102, the orderingis implemented as a matrix of dependencies in ordering controller 102;direct memory accesses (DMAs) with the relaxed ordering do not honor orhave any ordering dependencies; direct memory accesses (DMAs) with therelaxed ordering set ordering dependencies for other (i.e., succeeding)commands; regularly ordered commands (for which no unordered and norelaxed ordered indications are set) are ordered behind relaxed commands(which have relaxed ordered bits set in the relaxed ordering matrix).Therefore, even though the first data packets (for example, the payloadpackets) and the second data packets (for example, the signalingpackets) are sent back-to-back, ordering between the first data packets(for example, the payload packets) and the second data packets (forexample, the signaling packets) is guaranteed.

Ordering controller 102 has a set of Finite State Machines (FSMs) tohandle the processing of incoming commands. Each FSM handles a singlecommand at a time. Each incoming command is assigned to one of the FSMs.

One benefit of the present invention is latency reduction in a shortdistance coupling environment. The reason of the latency reduction isthat coupling device 101 does not wait for completions of the first datapackets (for example, the payload packets) and sends the second datapackets (for example, the signaling packets) back-to-back. Anotherbenefit of the present invention is that the implementation of thepresent invention requires no design change for coupling device 101.

Before detailed description of the embodiment of the present inventionis given, two current implementations for coupling transactions aredescribed with reference to FIG. 2, FIG. 3, and FIG. 4.

FIG. 2 is a diagram showing current implementation 200 for couplingtransactions, in accordance with an example of the first currentimplementation. In current implementation 200, the coupling device sendsin all packets, including payload packets and signaling packets, into amemory subsystem and requests all the packets to be ordered. As shown inthe example illustrated in FIG. 2, the coupling device sends payloadpacket P1 (denoted by numeral 201) to the ordering controller, and theordering controller sends payload packet P1 (denoted by numeral 201′) tothe memory subsystem. In response to receiving payload packet P1, thememory subsystem sends response R1 (denoted by numeral 206). Next, inresponse to receiving response R1 (denoted by numeral 206′), thecoupling device sends payload packet P2 (denoted by numeral 202) to theordering controller and the ordering controller sends payload packet P2(denoted by numeral 202′) to the memory subsystem. In response toreceiving payload packet P2, the memory subsystem sends response R2(denoted by numeral 207). In response to receiving response R2 (denotedby numeral 207′), the coupling device sends payload packet P3 (denotedby numeral 203) to the ordering controller and the ordering controllersends payload packet P3 (denoted by numeral 203′) to the memorysubsystem. In response to receiving payload packet P3, the memorysubsystem sends response R3 (denoted by numeral 208) to the couplingdevice. The coupling device receives response R3 (denoted by numeral208′). After a certain number of payload packets are sent and theircorresponding responses are received, the coupling device sends the n-thpayload packet Pn (denoted by numeral 204) to the ordering controllerand the ordering controller sends payload packet Pn (denoted by numeral204′) to the memory subsystem. In response to receiving payload packetPn, the memory subsystem sends response Rn (denoted by numeral 209) tothe coupling device. Finally, in response to receiving response Rn(denoted by numeral 209′), the coupling device sends signaling packet S(denoted by numeral 205) to the ordering controller and the orderingcontroller sends signaling packet S (denoted by numeral 205′) to thememory subsystem.

Current implementation 200 shown in FIG. 2 is implemented as a matrixshown in FIG. 2. FIG. 3 is example ordering matrix 300 for first currentimplementation 300 shown in FIG. 2, in accordance with an example offirst current implementation 200. By way of example, FIG. 3 shows anordering matrix of 4×4 only. Each column and each row of the matrixrepresents a FSM. A set bit in the matrix means: FSM in column X needsto wait for FSM in row Y to finish before FSM X can continue. Orderingmatrix 300 has only active entries for FSMs working on ordered commandsand does not know anything about FSMs working on unordered commands;this means that a command with ordering indication can overtake anunordered operation because the matrix has no information about theunordered commands. Ordering matrix 300 gives the sequence of executingcommands: row (denoted by numeral 301) of ordering matrix 300 waits forcolumn (denoted by numeral 302) to finish. In ordering matrix 300, thesequence of executing commands is A then B, then D, and then C. Column Ahas no X (no bit is set), so command A does not need to wait for othercommands. Column B has an X (a bit is set) in row A, so command B waitsuntil command A is done and the X is deleted. Column D has an X (a bitis set) in each of row A and row B, so command D until commands A and Bare completed. The command C is the last one to finish.

In current designs, the ordering controller maintains an ordering matrix(such as ordering matrix 300) has a size of N×N, where N is the numberof packets that are active in the ordering controller at a given moment.A bit being set in the row M of the column N means that the FSM (finitestate machine) N has an ordering dependency towards the FSM M (meaningthat N has to be ordered behind M). In addition, the ordering controllermaintains a table with a depth of N, where each bit indicates whetherthe FSM N is busy with an active ordered command. In the currentimplementation, when a command is received and an unordered bit is set,neither the ordering matrix nor the active ordered command table areupdated. In the current implementation, when a command is received forthe FSM N and the unordered bit is not set, the active ordered commandtable is copied into column N of the ordering matrix and the bit N inthe active ordered command matrix is set. When a command M completes,the row M in the ordering matrix is reset, as well as the bit M in theactive ordered command table.

FIG. 4 is a diagram illustrating current implementation 400 for couplingtransactions, in accordance with an example of current implementation400. In implementation 400, in order to optimize throughput, thecoupling device sends in payload packets associated with a couplingmessage back-to-back, with an indication to the memory subsystem not toimpose ordering on the payload packets. The coupling device waits forall responses to be received for the payload packets. When all theseresponses have been received, the signaling packets, which is with theordering flag being set, are sent in. Waiting for all payload packets tobe fully completed before being able to send in a signaling packet leadsto an additional latency penalty for an individual message. FIG. 4 showsthat the coupling device sends all payload packets unordered and waitsuntil all of the payload packets are complete before sending in thesignaling packets. As shown in the example illustrated in FIG. 4, thecoupling device sends payload packet P1 (denoted by numeral 401),payload packet P2 (denoted by numeral 402), payload packet P3 (denotedby numeral 403), . . . , and payload packet Pn (denoted by numeral 404)to the ordering controller and the ordering controller sends payloadpackets (denoted by numerals 401′, 402′, 403′, and 404′) to the memorysubsystem. The coupling device sends the payload packets in an unorderedmode. In response to receiving payload packets, the memory subsystemsends response R2 (denoted by numeral 406) corresponding to payloadpacket P2 (denoted by numeral 402), response R1 (denoted by numeral 407)corresponding to payload packet P1 (denoted by numeral 401), response Rn(denoted by numeral 408) corresponding to payload packet Pn (denoted bynumeral 404), and response R3 (denoted by numeral 409) corresponding topayload packet P3 (denoted by numeral 403). In response to receiving allthe responses (denoted by numerals 406′, 407′, 408′, and 409′), thecoupling device sends signaling packet S (denoted by numeral 405) to theordering controller and the ordering controller sends signaling packet S(denoted by numeral 405′) to the memory subsystem. The memory subsystemreceive the signaling packet (denoted by numeral 405″).

Now, detailed description of the embodiment of the present invention isprovided as follows, with reference to FIG. 5 and FIG. 6.

FIG. 5 is a diagram illustrating implementation 500 for couplingtransactions, in accordance with one embodiment of the presentinvention. FIG. 5 shows an example of the implementation in accordancewith the present invention. As shown in FIG. 5, the coupling devicesends payload packets P1, P2, P3, . . . , and Pn (denoted by numerals501, 502, 503, and 504) back-to-back to the ordering controller. Same asin implementation 400, the unordered attribute is set for the payloadpackets. Unlike implementation 400, without waiting for all theresponses to be received, the coupling device sends in signaling packetS (denoted by numeral 505) back-to-back to the ordering controller.Signaling packet S (505) is with ordering requested.

In response to receiving payload packets P1, P2, P3, . . . , and Pn(denoted by numerals 501′, 502′, 503′, and 504′), the orderingcontroller sends payload packets P1, P2, P3, . . . , and Pn to thememory subsystem. The ordering controller sends payload packets in anarbitrary order. The ordering controller also receives signaling packetS (denoted by numeral 505′); however, the ordering controller does notsend signaling packet S (denoted by numeral 505′) to the memorysubsystem until responses for all of payload packets P1, P2, P3, . . . ,and Pn are received.

In response to receiving payload packets P1, P2, P3, . . . , and Pn(denoted by numerals 501′, 502′, 503′, and 504′), the memory subsystemsends response R2 (denoted by numeral 506) corresponding to payloadpacket P2 (denoted by numeral 502), response R1 (denoted by numeral 507)corresponding to payload packet P1 (denoted by numeral 501), response Rn(denoted by numeral 508) corresponding to payload packet Pn (denoted bynumeral 504), and response R3 (denoted by numeral 509) corresponding topayload packet P3 (denoted by numeral 503).

In response to that all the responses (denoted by numerals 506′, 507′,508′, and 509′) are received, the ordering controller sends signalingpacket S (denoted by numeral 505′) to the memory subsystem. Thus, thememory subsystem receives signaling packet S (denoted by numeral 505″)after all payload packets R1, R2, R3, . . . , and Rn are received. Asshown in FIG. 5, before response R3 (denoted by 509′) is received by thecoupling device, the ordering controller already sends signaling packetS (denoted by numeral 505′) to the memory subsystem. Therefore, comparedto implementation 400 shown in FIG. 4, implementation 500 in the presentinvention reduces the latency for coupling messages.

In the implementation in accordance with the present invention, acoupling device (for example, an integrated cluster bus (ICB)) continuesto send in the first data packets (for example, the payload packets)with the unordered attribute being set back-to-back; the coupling devicealso sends in the second data packets (for example, the signalingpackets) back-to-back with ordering requested, instead of delaying thesecond data packets (for example, the signaling packets) and waitinguntil all the responses are received for all the first data packets (forexample, the payload packets). An ordering controller (for example, aPCI bus controller (PBC)) introduces a new mode for handling theunordered indication, i.e., changing its semantics to a relaxed orderingindication. While unordered commands (for example, the payloadoperations) continue to not have an ordering relation towards eachother, ordered commands (for example, the signaling operations) waituntil all preceding commands (for example, payload operation) havecompleted, even if these preceding commands have the unordered flag set.Therefore, the ordering is guaranteed on a high level but the latencypenalty in current implementations is removed. The latency penalty inthe current implementations is caused by waiting for all payload packetsto be fully completed and their completions to be signaled to thecoupling device.

The present invention introduces an additional mode that can be enabledin a register in the ordering controller (for example, the PCI buscontroller (PBC)). The mode leads to unordered operations being orderedin a relaxed ordering mode instead of no order at all. In terms of theexisting ordering infrastructure, no change applies to a command beingreceived with an unordered bit not set; this means that, with respect tosetting the unordered bit, the coupling device (for example, theintegrated cluster bus (ICB)) behaves exactly the same way as in currentimplementation 400 shown in FIG. 4. When command M is received with theunordered bit being set, instead of updating neither the ordering matrixnor the table of the active commands, only the ordering matrix is notupdated. Even when the unordered bit is set for command M, thecorresponding bit in the table of active ordered entries is set. Incurrent implementation 400 shown in FIG. 4, an unordered command was notreflected in the ordering controller at all; in the present invention,the new bit is set to indicate that other commands need to wait forcommand M. In the present invention, when command M comes in, a bit M isneeded to be set in a relaxed order matrix in the ordering controller,which indicates that other commands coming in later need to wait forcommand M. As a result, subsequent ordered commands (which come in aftercommand M and have the unordered bit not set) are ordered towards theunordered commands, while the unordered commands are still unorderedtowards each other and other ordered commands. A relaxed command (whosebit is set in the relaxed order matrix) can overtake other commands butmust not be overtaken by other commands which are not relaxed.

FIG. 6 is an example of relaxed ordering matrix 600 for implementation500 shown in FIG. 5, in accordance with one embodiment of the presentinvention. By way of example, FIG. 6 shows relaxed ordering matrix 600with a dimension of 4. The present invention introduces a new mode forhandling the unordered indication—a relaxed ordering indication. In FIG.6, 4×4 matrix 601 in relaxed ordering matrix 600 indicates a command(e.g., command C) needs to wait for other commands (e.g., commands A, B,and D), RO (relaxed order) column 602 in relaxed ordering matrix 600indicates the other commands need to wait for the command. In therelaxed ordering matrix, there is a new bit field—RO column 602. The newbit field (RO column 602) indicates that FSM is in relaxed ordering (RO)mode. As shown in FIG. 6, when a new FSM C is used without a RO bit set,all FSMs in RO column 602 (e.g., commands A, B, and D) each having a RObit set are completed before the new FSM C can continue. With therelaxed ordering mode, commands A, B, and D are executed in an arbitraryorder, command C waits until commands A, B, and D are finished. In theembodiment of the present invention, for example, commands A, B, and Dare for the payload operations (shown in FIG. 4) and command C is forthe signaling operation (shown in FIG. 5).

Based on the foregoing, a computer system with a configurable orderingcontroller for coupling transactions has been disclosed. However,numerous modifications and substitutions can be made without deviatingfrom the sprit and scope of the present invention. Therefore, thepresent invention has been disclosed by way of examples and notlimitation.

What is claimed is:
 1. A method for coupling transactions with aconfigurable ordering controller in a computer system, the methodcomprising: sending, by a coupling device, payload packets with anunordered attribute being set to an ordering controller; sending, by thecoupling device, a signaling packet with requested ordering to theordering controller, back-to-back after the payload packets, before thecoupling device receives from a memory subsystem responses indicatingthat the memory subsystem has received the payload packets; and sending,by the ordering controller, the payload packets to the memory subsystemin a relaxed ordering mode which is enabled in a register in theordering controller, wherein the ordering controller is in the relaxedordering mode such that the ordering controller sends the payloadpackets to the memory subsystem in an arbitrary order and sends thesignaling packet to the memory subsystem after the ordering controllerhas sent all of the payload packets to the memory subsystem.
 2. Themethod of claim 1, wherein the relaxed ordering mode is implemented as arelaxed ordering matrix of dependencies in the ordering controller. 3.The method of claim 2, wherein the relaxed ordering matrix comprises amatrix indicating that execution of commands for the signaling packetneeds to wait for execution of commands for the payload packets.
 4. Themethod of claim 2, wherein the relaxed ordering matrix further comprisesa relaxed order column indicating the relaxed ordering mode for thepayload packets.
 5. The method of claim 4, wherein bits in the relaxedorder column are set for commands for the payload packets, and whereinthe bits in the relaxed order column are not set for commands for thesignaling packet.
 6. The method of claim 5, wherein the commands for thepayload packets are executed in the arbitrary order.
 7. The method ofclaim 5, wherein execution of the commands for the signaling packetwaits until execution of the commands for the payload packets arecompleted.
 8. The method of claim 1, wherein direct memory accesses withthe relaxed ordering mode do not have ordering dependencies.
 9. Themethod of claim 1, wherein direct memory accesses with the relaxedordering mode set ordering dependencies for other succeeding commands.10. The method of claim 1, wherein commands without the unorderedattribute and without the relaxed ordering mode are ordered behindcommands with the relaxed ordering mode.